Systems and methods of facial and body recognition, identification and analysis

ABSTRACT

Systems and methods for learning and recognizing features of an image are provided. A point detector identifies points in an image where there are two-dimensional changes. A geometric feature evaluator overlays at least one mesh on the image and analyzes geometric features on the at least one mesh. An internal calibrator transforms data from the point detector and the geometric feature evaluator into a three-dimensional point figure of the image, and a depth evaluator determines a final shape of the image. A three-dimensional object model of the image is constructed. The image could be a human face or body. An artificial intelligence unit learns and identifies a user&#39;s facial features including skull size, distance between eyes, and bone structure and body features including skeleton shape and body size. Exemplary systems and methods can construct and learn features of a human face based on a partial view where part of the face is covered. Systems and methods can unlock a mobile device based on recognition of the features of the user&#39;s face.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 17/212,235, filed Mar. 25, 2021, with is anon-provisional of and claims priority to U.S. Patent Application Ser.No. 63/147,326, filed Feb. 9, 2021, each of which is hereby incorporatedby reference herein in its entirety.

FIELD

The present disclosure relates to systems and methods for learning andrecognizing features of an image such as a human face and unlockingfunctions for any computer or smartphone screen based on facial and bodyrecognition, including covered face or body.

BACKGROUND

Many mobile devices such as smartphones include a facial identification(ID) recognition system that learns the user's facial features andunlocks the phone upon its user's face image. Due to the COVID-10pandemic, in most public places a face cover or facemask is required.However, the presence of a face cover or mask interrupts the facerecognition unlocking feature on most mobile devices.

Accordingly, there is a need for a system and method that can learn andrecognize a partially covered human face. There is a need for a systemand method that can unlock a mobile device based on recognition of apartially covered human face.

SUMMARY

The present disclosure, in its many embodiments, alleviates to a greatextent the disadvantages of known devices, systems, and methods byproviding an AI-based computer vision system and method to lock andunlock a mobile device such as a smartphone with or without face (mask)coverage. The user trains the system once without a face cover (mask).After the initial training, the system is capable of identifying theuser's facial features with or without a face cover or facemask and iscapable of locking or unlocking it. In addition, the system can betrained once for the user's body's features. After learning the user'sbody's features, the system can monitor bodily changes like weight gainand other changes, alerting the user in real time.

Exemplary embodiments of a system for learning and recognizing featuresof an image comprise at least one point detector, at least one geometricfeature evaluator, at least one internal calibrator, and at least onedepth evaluator. The point detector identifies points in an image wherethere are two-dimensional changes. The geometric feature evaluatoroverlays at least one mesh on the image and analyzes geometric featureson the at least one mesh. The internal calibrator transforms data fromthe point detector and the geometric feature evaluator into athree-dimensional point figure of the image. The depth evaluatordetermines a final shape of the image.

In exemplary embodiments, the point detector and the geometric featureevaluator identify points based on geodesic distance between vertices inthe mesh. The geometric feature evaluator may use stereo vision toperform its tasks. The two-dimensional changes may comprise one or moreof: corners, junctions, and vertices. In exemplary embodiments, thesystem constructs a three-dimensional object model of the image. Thesystem is capable of constructing a three-dimensional object model ofthe image from a partial view of the image.

In exemplary embodiments, the image is of a human face or body. Thesystem may further comprise an artificial intelligence unit configuredto learn a user's facial and body features. In exemplary embodiments,the system is housed in a mobile device and is configured to lock orunlock the mobile device upon identification of the user's facial orbody features. The system may further comprise a neural network.Exemplary embodiments include an expert system configured to read datafrom the neural network and identify unique features of a user's face orbody and map the unique features into a database. The expert systemcomputes physical relations and ratios of unique facial and bodyfeatures including and not limited to distance and depth.

Exemplary computer-implemented methods of learning and recognizingfeatures of an image comprise identifying points in an image where thereare two-dimensional changes, overlaying at least one mesh on the imageand analyzing geometric features on the at least one mesh, transformingdata relating to the points and geometric features into athree-dimensional point figure of the image, and determining a finalshape of the image. The points may be identified based on geodesicdistance between vertices in the mesh. The two-dimensional changescomprise one or more of: corners, junctions, and vertices. The geometricfeatures may be analyzed using stereo vision. Exemplary methods furthercomprise constructing a three-dimensional object model of the image.

In exemplary embodiments, the image is of a human face or body andmethods further comprise learning features of a user's face or body.Exemplary methods comprise identifying the features of the human faceand unlocking a mobile device based on recognition of the features ofthe user's face. In exemplary embodiments the constructing step isperformed based on a partial view of the image and the learning isperformed based on a partial view of the user's face. The recognitionand unlocking may be performed based on a partial view of the featuresof the user's face. Exemplary methods further comprise storing as areference data relating to the features of the user's face.

Accordingly, it is seen that systems and methods of learning andrecognizing features of an image are provided. These and other featuresof the disclosed embodiments will be appreciated from review of thefollowing detailed description, along with the accompanying figures inwhich like reference numbers refer to like parts throughout.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects of the disclosure will be apparent uponconsideration of the following detailed description, taken inconjunction with the accompanying drawings, in which:

FIG. 1 is a process flow diagram of an exemplary embodiment of a systemand method for learning and recognizing features of an image inaccordance with the present disclosure;

FIG. 2 is a front view of an exemplary embodiment of a system and methodfor learning and recognizing features of an image in accordance with thepresent disclosure;

FIG. 3 is a schematic of an exemplary embodiment of a system and methodfor learning and recognizing features of an image using a neural networkfor facial mapping in accordance with the present disclosure;

FIG. 4 is a schematic of an exemplary embodiment of a system and methodfor learning and recognizing features of an image by generating athree-dimensional point figure in accordance with the presentdisclosure;

FIG. 5 is a side view of the embodiment shown in FIG. 4 ;

FIG. 6 is a process flow diagram of an exemplary embodiment of a displayand graphical user interface in accordance with the present disclosure;

FIG. 7 is a perspective view of an exemplary embodiment of a method oflearning and recognizing features of an image of a user's face inaccordance with the present disclosure;

FIG. 8 is a schematic of an exemplary embodiment of a method of learningand recognizing features of an image of a user's face in accordance withthe present disclosure;

FIG. 9 is a schematic of an exemplary embodiment of a method of learningand recognizing features of an image of a user's body in accordance withthe present disclosure;

FIG. 10 is a schematic of an exemplary embodiment of a method oflearning and recognizing features of an image of a user's body inaccordance with the present disclosure;

FIG. 11A is a front view of an exemplary embodiment of a method ofrecognizing features of a partially covered image of a user's face inaccordance with the present disclosure;

FIG. 11B is a front view of an exemplary embodiment of a method ofrecognizing features of a partially covered image of a user's face inaccordance with the present disclosure;

FIG. 12 is a schematic diagram of an exemplary embodiment of anartificial intelligence system in accordance with the presentdisclosure;

FIG. 13 is a process flow diagram of an exemplary embodiment of a methodof learning and recognizing features of an image using pixelation andAI-based computer vision analysis in accordance with the presentdisclosure;

FIG. 14 is a schematic of an exemplary embodiment of a method oflearning and recognizing features of an image using pixelation andAI-based computer vision analysis in accordance with the presentdisclosure;

FIG. 15 is a schematic of an exemplary embodiment of a method oflearning and recognizing features of an image using pixelation andAI-based computer vision analysis in accordance with the presentdisclosure;

FIG. 16 is a schematic of an exemplary embodiment of a method oflearning and recognizing features of an image using AI-based computervision analysis in accordance with the present disclosure

FIG. 17 is a schematic of an exemplary embodiment of a method oflearning and recognizing features of an image using AI-based computervision analysis in accordance with the present disclosure;

FIG. 18 is a schematic of an exemplary embodiment of a method oflearning and recognizing features of an image using AI-based computervision analysis in accordance with the present disclosure;

FIG. 19 is a perspective view of an exemplary embodiment of a system andmethod of learning and recognizing features of an image usingvectorization and deep learning in accordance with the presentdisclosure;

FIG. 20A is a front view of an exemplary embodiment of a mobile deviceLOCK/UNLOCK feature in accordance with the present disclosure; and

FIG. 20B is a front view of an exemplary embodiment of a mobile deviceLOCK/UNLOCK feature in accordance with the present disclosure.

DETAILED DESCRIPTION

In the following paragraphs, embodiments will be described in detail byway of example with reference to the accompanying drawings, which arenot drawn to scale, and the illustrated components are not necessarilydrawn proportionately to one another. Throughout this description, theembodiments and examples shown should be considered as exemplars, ratherthan as limitations of the present disclosure.

As used herein, the “present disclosure” refers to any one of theembodiments described herein, and any equivalents. Furthermore,reference to various aspects of the disclosure throughout this documentdoes not mean that all claimed embodiments or methods must include thereferenced aspects. Reference to materials, configurations, directions,and other parameters should be considered as representative andillustrative of the capabilities of exemplary embodiments, andembodiments can operate with a wide variety of such parameters. Itshould be noted that the figures do not show every piece of equipment,nor the materials, configurations, and directions of the variouscircuits and communications systems.

In the present disclosure and its embodiments systems are providedincluding a mobile application and computer software. Exemplaryembodiments learn a user's facial and body features by one-time usertraining. The user places the smartphone or other mobile device in frontof his or her face and body, and the system learns the facial and bodyfeatures. Based on this information, even if the user covers his or herface with a face mask, or any other type of cloth or covering, thesystem can identify him or her even with the face cover partiallyobscuring the face.

The same applies to the user's body features. The computer visionsoftware learns the user's facial and body features once. Then it canidentify the user's face or body features when they are fully coveredwith or without clothing and face cover. The system can identify auser's facial and body's changes, like weight gain or similar changes,alerting the user in real time. This feature can be used for deviceLOCK/UNLOCK, health watcher, clothing estimation and similarapplications.

In exemplary embodiments, a fast and robust system detects, identifiesand localizes human body parts within a software application. Theinformation can be used as preprocessing for facial and body IDrecognition and LOCKING/UNLOCKING algorithms. Disclosed embodiments canbe used for smartphone and computer security LOCK/UNLOCK features basedon facial and/or full-body features. They also can be used for trackingor surveillance.

Referring to FIGS. 1-3 , an exemplary system 1 for learning andrecognizing features of an image is illustrated. A detection andidentification feature 2 performs face extraction 3 and body extraction4 by finding specific, predefined points in the image 8 and computingdescriptors for the local features around them. More particularly, oneor more point detectors 10 identify points 12 in an image where thereare two-dimensional changes 14. These point detectors 10 identify pointsin the image for which the signal changes two-dimensionally, e.g., atcorners, junctions, and vertices. This method applies for facial andbody parts.

In addition, computer vision features are applied directly to the giventhree-dimensional data to develop detectors for locally interestingpoints 112. Exemplary embodiments apply vision-based interest pointdetectors 10 on depth images 108 to construct 3D object models 111 in anunsupervised fashion from partial views. The imaging produces coloredareas of interest to identify the location of the face and body featuresusing computer vision techniques known in the art. Exemplary embodimentsperform virtual face model construction 46 and virtual body modelconstruction 48 when classification 45 of the face and body features aresuccessful.

Turning also to FIGS. 4 and 5 , another feature considers spectralgeometric features on triangular meshes 16 and recognizes pointinggestures using stereotypical vision. In exemplary embodiments, one ormore geometric feature evaluators 6 overlay at least one mesh 16 on theimage 8 and analyze geometric features 15 on the at least one mesh.Exemplary embodiments identify points of interest 12 based on geodesicdistance between vertices 24 in a mesh 16.

This analysis for the points of interest detector has the advantage ofproviding a stable estimate of a person's shape and pose, which can beused to digitize his or her image 8, 108 prior to a figure'sconstruction. Based on these features, exemplary systems and methodsfurther categorize a person's features and are able to digitally“remove” facial and body covers, reconstructing the person's face andbody for identification. As discussed in more detail herein, uponreconstruction of a person's features a LOCK/UNLOCK feature can beactivated which is a straightforward method. The LOCK/LOCK feature 220is shown in FIGS. 20A and 20B.

Distance measurements are transformed into a 3D point FIG. 120 using aninternal calibration feature. More particularly, one or more internalcalibrators 18 transform data from the point detectors 10 and thegeometric feature evaluators 6 into a three-dimensional point FIG. 120of the image 8, 108. One or more depth evaluators 22 determine the finalshape of the image 8, 108. The roles of point detectors 10, calibrators18, and depth evaluators 22 are best seen in FIG. 13 .

Exemplary embodiments may include one or more of the following features.The system may include an artificial intelligence system 50 to learn theuser's facial and body features, including but not limited to, skullsize, distance between the eyes, and bone structure. In addition,exemplary systems and methods can learn the user's body's features likeskeleton shape, body size, special and personal features. After thetraining, the system can identify the user's body and/or facial featuresfor the purpose of locking or unlocking a smartphone, personal computer,and other apparatus.

Exemplary implementations of the described techniques may includehardware, a method or process, or computer software on acomputer-accessible medium. Disclosed systems and methods may alsoinclude mobile application software and hardware. For security purposesthe user still has the option to enter a passcode to unlock his or hermobile device. Exemplary embodiments include a processor for executinginstructions and a memory for storing executable instructions. Theprocessor executes the instructions to perform various functions.Another medium is a smartphone application to store and process the datafor a wide variety of purposes. In exemplary embodiments, the smartphoneapplication communicates with a backend program that runs on a computerserver to learn, store, and process the data.

As shown in FIG. 6 , in exemplary embodiments, a display 26 on agraphical user interface (GUI) 28 comprises an input interface 30 thatreceives a user-selected file having at least one image 8. An artificialintelligence system 50 receives the user's facial and body imagingcontent. In exemplary embodiments, the systems analyze the contentaccording to the user's personal facial and body features and convert,on-demand, the user-selected content into an internal digital data fileby combining the pre-learned user's facial and body information andadditional, required features like LOCK/UNLOCK features, shown in FIGS.20A and 20B. The system performs a user's facial and body study andrecognition that is stored as a reference in a face database 42 and bodydatabase 44, respectively, shown in FIG. 1 .

In operation, the user trains the system to identify his or her face byholding the mobile device 32 in front of his or her face once, as bestseen in FIGS. 7 and 8 and taking a picture using the camera 222 in thedevice. As shown in FIGS. 9 and 10 , the user may also train the systemto identify his body by holding the mobile device 32 in front of hisbody. As discussed above, exemplary processes include identifyingspecific points 12 in the image of the face and/or body where there aretwo-dimensional changes 14 such as corners, junctions, vertices, etc.and/or other points of interest 112.

Another process step may include overlaying a mesh 16 on the image 8,108 of the face and/or body and analyzing geometric features 15 on themesh. The process next performs the steps of transforming data from theidentified points 12 and points of interest 112 and the geometricfeature analysis into a three-dimensional point FIG. 120 of the faceand/or body image 8, 108, determining a final shape of the image, and insome embodiments, constructing a three-dimensional object model 108 ofthe face and/or body. As shown in FIGS. 11A-11B, after these steps, theuser's face can be identified and recognized with or without a covering40 obscuring it. Similarly, the user's body can be recognized with theuser wearing any type of clothing.

With reference to FIG. 12 , in exemplary embodiments incorporatingAI-based computer vision analysis, proprietary image recognitionprocesses are utilized. An overview of an exemplary AI system isillustrated in FIG. 12 . The process of real time picture capture 101includes neural network analysis 52 and use of a dataset picture 54.Registering 56 of the body and/or face illustration model is followed bythe query 58 whether the image model presents in the face image modelsdatabase 42 and/or body image models database 44. If no, the body orface image is input into its respective database. If yes, then theprocess proceeds to the modeling picture features step 60. A databasecomparison 62 is performed, and if there is matching 64 of the imagemodel record, the identification is complete 66.

In exemplary embodiments, an expert system provides ratios and relationsanalysis between unique key pointers in a human face and body. Picturesthat are taken without mask/clothing are mapped by the expert system toidentify vectorials based on key features' relations and ratios. Theexpert system uses the neural network data and identifies unique keyfacial and body points of interest. These unique features are mapped as2D and 3D databases to be later used as key identifiers to identifypeople with face or body coverings. The expert system can computephysical relations and ratios of unique facial and body features likedistances, e.g., the distance between the nose and mouth on the rightside of the face relative to the distance between the nose and mouth onthe left side of the face, depth level, e.g., the depth between the lefteye relative to the depth of the right eye. The expert system input isthe neural network data and the output is a vectorial map of unique, outof the ordinary, or stick-out facial and body features. The process canbe done for all facial and body views. The more views provided, thebetter results can be achieved. The facial and body views are front,back, and sides.

Turning to FIGS. 13-19 , exemplary image pixelation methods will bedescribed. The image pixelation process 201 includes not onlyfacial/body detection 202 but also classifier training 205. If the bodyis detected 207, the body features are recorded. If the body is notdetected, the facial body detection step 202 may be repeated. Theprocess will also attempt to detect 211 the user's face. If the face isnot detected, the facial body detection step 202 may be repeated. Whenthe face is detected 211, the process engages in deep learning 213,which could be for the body and/or the face.

Body and/or face vectorization 215 may then be performed, a functionillustrated in more detail in FIG. 19 . The calibrators 18 contribute tothis part of the process and may assist in transforming image data intoa three-dimensional point figure of the image. As discussed above, pointdetectors 10 generate key points in the image where there aretwo-dimensional changes, and geometric feature evaluators 6 analyzegeometric features on a mesh overlayed on the image. Then, using the keypoints and geometric feature analysis, depth evaluators 22 determine thefinal shape of the image and the recognition process is completed 219.The process may include recognition training 217 based on the deeplearning, vectorization and point detection functions.

In a first phase of AI analysis, illustrated in FIG. 14 , highresolution, pixelation-based facial mapping is performed using a neuralnetwork 34 and associate neural network analysis 52. A second phaseshown in FIG. 3 may perform low resolution, pixelation-based facialmapping using the neural network. In a third phase, portrait wide facialmapping is performed using the neural network 34, as shown in FIG. 15 .As illustrated in FIG. 16 , exemplary methods may perform sidewaysfacial mapping using an expert system in phase four of the analysis. Inphase five, shown in FIG. 4 , biometric facial mapping is done using theexpert system. Phase six may comprise a human body style study 36 basedon AI vector mapping, as illustrated in FIG. 17 . Finally, referring toFIG. 18 , in phase seven the analysis may include the step ofidentifying the clothing and accessories 38 of the user based on AIvector mapping. The user can be identified with or without clothing. Ifthe user so desires, he or she can use the AI analysis to detect changesin her body and provide health alerts in the event of body changesindicating health problems.

Thus, it is seen that systems and methods for learning and recognizingfeatures of an image such as a human face and/or body are provided. Itshould be understood that any of the foregoing configurations andspecialized components or connections may be interchangeably used withany of the systems of the preceding embodiments. Although illustrativeembodiments are described hereinabove, it will be evident to one skilledin the art that various changes and modifications may be made thereinwithout departing from the scope of the disclosure. It is intended inthe appended claims to cover all such changes and modifications thatfall within the true spirit and scope of the present disclosure.

What is claimed is:
 1. A system for learning and recognizing features ofan image, comprising: at least one point detector identifying points inan image where there are two-dimensional changes; at least one geometricfeature evaluator overlaying at least one mesh on the image andanalyzing geometric features on the at least one mesh; at least oneinternal calibrator transforming data from the point detector and thegeometric feature evaluator into a three-dimensional point figure of theimage; an artificial intelligence unit configured to learn and identifya user's facial features including skull size, distance between eyes,and bone structure, and the user's body features including skeletonshape and body size; and at least one depth evaluator determining afinal shape of the image.
 2. The system of claim 1 wherein the image isof a human face and a human body.
 3. The system of claim 1 wherein thepoint detector and the geometric feature evaluator identify points basedon geodesic distance between vertices in the mesh.
 4. The system ofclaim 1 wherein the geometric feature evaluator uses stereo vision. 5.The system of claim 1 wherein the two-dimensional changes comprise oneor more of: corners, junctions, and vertices.
 6. The system of claim 1wherein the system constructs a three-dimensional object model of theimage.
 7. The system of claim 6 wherein the system constructs athree-dimensional object model of the image from a partial view of theimage.
 8. The system of claim 1 wherein the system is housed in a mobiledevice and is configured to lock or unlock the mobile device uponidentification of the user's facial or body features.
 9. The system ofclaim 1 further comprising a neural network.
 10. A computer-implementedmethod of learning and recognizing features of an image, comprising:identifying points in an image where there are two-dimensional changes;transforming data relating to the points and geometric features into athree-dimensional point figure of the image; determining a final shapeof the image; constructing a three-dimensional object model of theimage; and registering the three-dimensional object model in an imagemodels database.
 11. The method of claim 10 further comprising queryingwhether the three-dimensional object model presents in the image modelsdatabase.
 12. The method of claim 10 wherein the image is of a humanface or body and further comprising learning features of a user's faceor body.
 13. The method of claim 12 further comprising identifying thefeatures of the human face and unlocking a mobile device based onrecognition of the features of the user's face.
 14. The method of claim10 wherein one or both of the identifying and constructing steps areperformed based on a partial view of the image.
 15. The method of claim13 wherein the recognition and unlocking are performed based on apartial view of the features of the user's face.
 16. The method of claim10 further comprising storing as a reference data relating to thefeatures of the user's face.
 17. A computer-implemented method oflearning and recognizing features of an image of a human face,comprising: performing image pixelation including high resolutionpixelation-based facial mapping and low resolution pixelation-basedfacial mapping; performing sideways facial mapping; performing biometricfacial mapping; identifying points in an image where there aretwo-dimensional changes; overlaying at least one mesh on the image andanalyzing geometric features on the at least one mesh; transforming datarelating to the points and geometric features into a three-dimensionalpoint figure of the image; and determining a final shape of the image.18. The method of claim 17 further comprising performing a study of ahuman body based on vector mapping.
 19. The method of claim 18 furthercomprising identifying clothing based on vector mapping.
 20. The methodof claim 18 further comprising identifying accessories based on vectormapping.